A Maintenance Planning Framework using Online and Offline Deep Reinforcement Learning

Cost-effective asset management is an area of interest across several industries. Specifically, this paper develops a deep reinforcement learning (DRL) solution to automatically determine an optimal rehabilitation policy for continuously deteriorating water pipes. We approach the problem of rehabilitation planning in an online and offline DRL setting. In online DRL, the agent interacts with a simulated environment of multiple pipes with distinct lengths, materials, and failure rate characteristics. We train the agent using deep Q-learning (DQN) to learn an optimal policy with minimal average costs and reduced failure probability. In offline learning, the agent uses static data, e.g., DQN replay data, to learn an optimal policy via a conservative Q-learning algorithm without further interactions with the environment. We demonstrate that DRL-based policies improve over standard preventive, corrective, and greedy planning alternatives. Additionally, learning from the fixed DQN replay dataset in an offline setting further improves the performance. The results warrant that the existing deterioration profiles of water pipes consisting of large and diverse states and action trajectories provide a valuable avenue to learn rehabilitation policies in the offline setting, which can be further fine-tuned using the simulator.Comment: Published Neural Comput & Applic (2023), 12 pages, 8 Figur

Deterioration modeling of sewer pipes via discrete-time Markov chains: A large-scale case study in the Netherlands

[For the latest version of this repository go to: https://gitlab.utwente.nl/fmt/degradation-models/dtmc_sewer_pipes.git] Sewer pipe network systems are an important part of civil infrastructure, and in order to find a good trade-off between maintenance costs and system performance, reliable sewer pipe degradation models are essential. In this paper, we present a large-scale case study in the city of Breda in the Netherlands. Our dataset has information on sewer pipes built since the 1920s and contains information on different covariates. We also have several types of damage, but we focus our attention on infiltrations, surface damage, and cracks. Each damage has an associated severity index ranging from 1 to 5. To account for the characteristics of sewer pipes, we defined 6 cohorts of interest. Two types of discrete-time Markov chains (DTMC), which we called Chain `Multi' and `Single' (where Chain `Multi'contains additional transitions compared to Chain `Single'), are commonly used to model sewer pipe degradation at the pipeline level, and we want to evaluate which suits better our case study. To calibrate the DTMCs, we define an optimization process using Sequential Least-Squares Programming to find the DTMC parameter that best minimizes the root mean weighted square error. Our results show that for our case study there is no substantial difference between Chain `Multi' and `Single', but the latter has fewer parameters and can be easily trained. Our DTMCs are useful to compare the cohorts via the expected values, e.g., concrete pipes carrying mixed and waste content reach severe levels of surface damage more quickly compared to concrete pipes carrying rainwater, which is a phenomenon typically identified in practice.This research has been partially funded by NWO under the grant PrimaVera (https://primavera-project.com) number NWA.1160.18.238, and has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 101008233

Upper limit to magnetism in LaAlO3/SrTiO3 heterostructures

Using polarized neutron reflectometry (PNR) we measured the neutron spin dependent reflectivity from four LaAlO3/SrTiO3 superlattices. This experiment implies that the upper limit for the magnetization induced by an 11 T magnetic field at 1.7 K is 2 emu/cm3. SQUID magnetometry of the superlattices sporadically finds an enhanced moment, possibly due to experimental artifacts. These observations set important restrictions on theories which imply a strongly enhanced magnetism at the interface between LaAlO3 and SrTiO3